Learning Methods to Generate Good Plans: Integrating HTN Learning and Reinforcement Learning

We consider how to learn Hierarchical Task Networks (HTNs) for planning problems in which both the quality of solution plans generated by the HTNs and the speed at which those plans are found is important. We describe an integration of HTN Learning with Reinforcement Learning to both learn methods by analyzing semantic annotations on tasks and to produce estimates of the expected values of the learned methods by performing Monte Carlo updates. We performed an experiment in which plan quality was inversely related to plan length. In two planning domains, we evaluated the planning performance of the learned methods in comparison to two state-of-the-art satisficing classical planners, FASTFORWARD and SGPLAN6, and one optimal planner, HSPF . The results demonstrate that a greedy HTN planner using the learned methods was able to generate higher quality solutions than SGPLAN6 in both domains and FASTFORWARD in one. Our planner, FASTFORWARD, and SGPLAN6 ran in similar time, while HSPF was exponentially slower.